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Q . Abstract 

o 

^^ ' We consider a distributed source coding system in which several observations are communicated to 

CIh, the decoder using hmited transmission rate. The observations must be separately coded. We introduce a 

^^ ■ robust distributed coding scheme which flexibly trades off between system robustness and compression 

p^ ' efficiency. The optimality of this coding scheme is proved for various special cases. 

Index Terms — CEO problem, common information, distributed source coding, multiple descriptions. 

C/3 ' 
_CJ 

I. Introduction 

There are many situations in which data collected at several sites must be transmitted to 

O ' a common point for subsequent processing. Via clever encoding techniques, it is possible to 

■^ , 

O , capitalize on the correlation between data received at the various sites even though each encoder 

\D 

O I operates with no or only partial knowledge of the data received at the other sites. Slepian and 

O ■ Wolf [1] proved a coding theorem for two correlated memory less sources with separate encoders. 



They dealt with the case where the decoder must reproduce two source outputs with arbitrary 
small error probability. Their results were extended to arbitrary number of discrete sources 
with ergodic memory and countably infinite alphabets by Cover [2]. Based on the results of 
Slepian and Wolf, Wyner and Ziv [3] extended rate-distortion theory to the case in which side 
information is present at the decoder. Berger [4] and Tung [5] generalized the Slepian- Wolf 
problem by considering general distortion criteria on the source reconstruction. The complete 
characterization of the rate-distortion region is unknown except for the special case where one of 
two source outputs must be reconstructed with an arbitrary small error probability and the other 
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must have an average distortion smaller than a prescribed level [6]. Oohama [7] studied the rate- 
distortion region for correlated memoryless Gaussian sources and squared distortion measures. 
He demonstrated that the inner bound of the rate-distortion region obtained by Berger and Tung is 
partially tight in the Gaussian case. Viswanath [8] characterized the sum-rate distortion function 
of Gaussian multiterminal source coding problem for a class of quadratic distortion metrics. 
A closely related problem, called the remote source coding problem or the CEO problem, has 
been studied in [9]-[13]. Oohama [14] derived the sum-rate distortion function for the quadratic 
Gaussian CEO problem when there are infinite encoders and the SNRs at all the encoders are 
identical. It was observed by Chen et al. [15] that Oohama's converse yields a tight upper 
bound on the sum-rate distortion function even when the number of encoders are finite. They 
also computed the achievable region for the general quadratic Gaussian CEO problem. Recently, 
Oohama [16] and Prabhakaran et al. [17] showed that this achievable region is indeed the rate- 
distortion region. 

Another important class of source coding problems is called multiple description problem. In 
the multiple description problem, the total available bit rate is split between (say) two channels 
and either channel may be subject to failure. It is desired to allocate rate and coded representations 
between the two channels, such that if one channel fails, an adequate reconstruction of the source 
is possible, but if both channels are available, an improved reconstruction over the single-channel 
reception results. This problem was posed by Gersho, Witsenhausen, Wolf, Wyner, Ziv and 
Ozarow in 1979. Early contributions to this problem can be found in Witsenhausen [18], Wolf, 
Wyner and Ziv [19], Ozarow [20] and Witsenhausen and Wyner [21]. The first general result was 
El Gamal and Cover's achievable region for two channels [22]. Ahlswede [23] showed that in 
the "no excess rate" case. El Gamal and Cover's region is tight. Zhang and Berger [24] exhibit 
a simple counterexample that shows El Gamal and Cover's region is not always tight in the case 
of an excess rate. Further results can be found in [25]-[32]. 

Distributed source coding problems of the Slepian-Wolf type and its extensions emphasize the 
compression efficiency of coding system but ignore the system robustness. A distributed source 
coding scheme which is optimal in the sense of compression efficiency can be very sensitive to 
the encoder failure, i.e., the performance of the whole system may degenerate dramatically when 
one of the encoders is subject to a failure. On the other hand, multiple description problem does 
consider the system robustness. But it is essentially a centralized source coding problem whose 
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Fig. 1. Model of robust distributed source coding system 



coding schemes in general can not be applied in the distributed source coding scenario. So it is 
of interest to study robust distributed source coding scheme, which is able to trade off between 
two important parameters: system robustness and compression efficiency. 



II. System Model and Problem Formulation 

Consider the distributed source coding system shown in Fig. 1. Let {X(t),Yi(t),Y2{t)}'^^ 
be temporally memoryless source with instantaneous joint probability distribution P(x, 1/1,1/2) 
on X X yi X y2, where X is the common alphabet of the random variables X(t) for t = 
1, 2, ■ ■ ■ , yi [i = 1,2) is the common alphabet of the random variables Fj(t) for t = 1, 2, ■ ■ ■ . 
{X(t)}^]^ is the target data sequence which can not be observed directly. Instead, two corrupted 
versions of {X(t)}^]^, i.e., {Yi^t)}"^-^ and {Y2{t)}'^^, are observed by encoder 1 and encoder 
2 respectively. Encoder i encodes a block y" = [|/j(l), ■ ■ ■ , yi{n)] of length n from its observed 



,(n) 



data using a source code cf'' = fEiiUi') of rate -log |Q"^|. Decoder i reconstructs the target 



Wi 



sequence a;" = [x(l),--- ,x{n)] by implementing a mapping fj^- : Q 



in) 



An) 



A'", 



1,2. 



Decoder 3 reconstructs the target sequence a;" = [x{l] 



,x{n)] by implementing a mapping 



An) 



An) 



in) 



Definition 1: The quintuple (_Ri, R2, Di, D2, -D3) is called achievable, if for any e > 0, there 
exists an no such that for all n > uq there exist encoders: 



An) 



An) 



m^yr^cr \og\cn<n{R,+e 



(")l 



1,2 



and decoders: 



fin) . Mn) 
J D,i ■ ^i 



An) . p{n) Mn) 
J D,3 • '-'1 ^ '-'2 



= 1,2 
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such that for Xf = f^^} (/g(Fr)) , ^ = 1, 2, and for X^ = f^^ (/g(n"), f^^^{Y- 

1 " 

-Ey^ d(X(t),Xi(t)) <Di + e i = 1,2,3. 



n 

t=l 



Here d{-, ■) : X x X ^ [0, rfmax] is a given distortion measure. 
Let Q denote the set of all achievable quintuples. 
Remark: 

1) Our model applies to many different scenarios such as the nonergodic link failures from 
some encoders to the decoder or the malfunction of some encoders; 

2) We restrict our treatment to the case of two encoders just for simplifying the notations. 
Most of our results can be extended in a straightforward way to the case of arbitrary 
number of encoders; 

Our model was first introduced by Ishwar et al. in [33]. An analogous problem called multilevel 
diversity coding has been studied in [34]-[37]. But it is a centralized source coding problem 
since all the encoders have the same observation. A distributed version of multilevel diversity 
coding was introduced in [38], where only the case of lossless source coding was treated. 

The rest of this paper is divided into four sections. Section III uses some examples to motivate 
the results. In Section IV, we first consider two different scenarios, namely, the centralized 
source coding and the distributed source coding, for which the corresponding coding schemes 
are established. Then we propose a unified approach by developing a coding scheme based on 
the idea of common information. In Section V, the case of correlated memoryless Gaussian 
observations and squared distortion measure is studied in detail. The inner bound and outer 
bound of the rate distortion region are established. We show that in various special cases the 
complete characterization of the rate distortion region is possible. Finally Section VI concludes 
the paper. 

III. Motivations and Examples 

Let Z^max = min^gg^ E,d(X, xq). Our problem reduces to the CEO problem if min(Di, D2) > 
Djnax and reduces the multiple description problem if there exist deterministic functions /j 
{i = 1, 2) such that X(t) = /j(Fj(t)) with probability one for t = 1, 2, ■ ■ ■ . So it is instructive 
to review the coding schemes for the CEO problem and multiple description problem. 
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For the CEO problem, the fidelity criterion is only imposed on the reconstruction of the target 
sequence at decoder 3. The largest known achievable rate distortion region for the CEO problem 
is the set' of (i?i, -R2, -D3) for which there exist random variables W^i, W2 jointly distributed with 
the generic source variables X, Yi and Y2 such that 

(i) Wi^Yi^ (X, F2, W2f and W2^Y2^ (X, Fi, Wi). 
(ii) i?i > I{Yv,Wi\W2),R2 > I{Y2;W2\Wi),Ri + R2 > I{YuY2;Wi,W2). 
(iii) There exist a function / : Wi x W2 ^ Af such that Ed{X, X) < D, where X = f{Wi, W2). 

The proof of the achievability of this rate distortion region is based on the idea of random 
binning. The main feature of the random binning coding scheme is outlined as follows: 

There are many bins at each encoder and many codewords in each bin. Instead of directly 
sending the codeword, each encoder sends the index of bin which contains the codeword that 
this encoder wants to reveal to the decoder. Upon receiving the indices of bins from all the 
encoders, the decoder picks one codeword from each bin such that these codewords are jointly 
typical. 

There are two important parameters for each encoder: the number of bins and the number 
of codewords. Roughly speaking, the number of bins determines the rate of the encoder while 
the number of codewords is associated with the description ability of the encoder. When the 
system is optimized in the sense of compression efficiency, the number of bins is minimized at 
each encoder if the number of its codewords is fixed (or equivalently, the number of codewords 
is maximized at each encoder if the number of its bins is fixed). Note: there exists a tradeoff 
between the maximum number of codewords at different encoders if the number of bins is fixed 
at each encoder (or equivalently, a tradeoff between the minimum number of bins at different 
encoders if the number of codewords is fixed at each encoder). But intuitively this optimization 
is achieved at the price of sacrificing the robustness of the whole system: if the decoder only 
receives the data from one of the encoders, then it may not be able to recover the correct codeword 
since the decoder only gets a bin index from one encoder and there are many codewords in that 
bin. Clearly, if there is only one codeword in each bin, then the decoder is able to recover the 
codeword as long as the bin index is received. Actually now the encoding scheme reduces to 



'By a timesharing argument, the convex hull of this region is also achievable. 

^A^B^C means A, B, and C form a Markov chain, i.e., A and C are independent conditioned on B. 



DRAFT 



6 SUBMITTED TO IEEE TRANSACTIONS ON INFORMATION THEORY 

the conventional lossy source encoding and the joint decoding scheme becomes the separate 
decoding. In general, we can improve the robustness of the distributed source coding system 
by reducing the number of codewords in each bin, which is a way to trade the compression 
efficiency for the system robustness. This is essentially the main idea of the robust distributed 
source coding scheme proposed by Ishwar, Puri, Pradhan and Ramchandran [33], which we 
will refer to as the IPPR scheme. The achievable rate distortion region of IPPR scheme for our 
model is the set of (i?i, R2, Di, D2, -D3) for which there exist random variables Wi, W2 jointly 
distributed with the generic source variables X, Yi and Y2 such that 
(i) Wi^Yi^ (X, F2, W2) and W2^Y2^ (X, Fi, Wi). 
(ii) Ri>I{Yi-Wi),R2>I{Y2;W2). 

(iii) There exist functions fi : Wi -^ X (i = 1, 2), and /s : Wi x W2 ^ A" such that 
m{X,Xi) < A {^ = 1,2,3), where X, = h{W,), % = ^(W^s) and X3 = /alW^i, W^s). 
We need the following definition before discussing the properties of the IPPR scheme. 
Definition 2: 

Dl{Ri,R2) = min{Di : {R,, R2, D,, D„,,^, D^,^) E Q}, 

D;{R,,R2) = mm{D2 : (i?i, i?2, A.ax, A, Anax) G Q}, 

D;{Ri,R2) = min{A : (i?i, i?2, Aiax, A.ax, A) e Q}. 

It is clear that D\{Ri,R2) does not depend on R2 and D2{Ri,R2) does not depend on Ri, so 
we shall denote them by Dl{Ri) and Dl{R2) respectively. Dl{Ri) and D|(i?2) are essentially 
the distortion-rate functions with noisy observations [39] [40], i.e., 

D*{Ri)= min Ed{X,Xi), i = l,2. 

XiG^:X—>-Yi^^Xi 
I{Y,;X,)<R, 

The IPPR scheme is of special interest in the sense that given rate tuple {Ri,R2), it can 
achieve Dl{Ri) and D|(i?2) at decoder 1 and decoder 2 respectively as shown by the following 
argument: 

Let W*{Ri),W2{R2) e X he the random variables jointly distributed with X,Yi and Y2 
such that W^{Ri) ^ Fi ^ {X,Y2,W^{R2)) and W^{R2) ^ Y2 ^ (X,Fi, W^*(i?i)) with 
I{Yf,W*{R,)) < R, and Ed{X,W*{R,)) = D*{R,),t = 1,2. 

By the IPPR scheme, (i?i, R2, DKRi), A(i?2), min^.;^ xA'^;t Ed{X, g{W^{Ri), 1^2*(^2)))) is 
achievable, where mmg.,xxx^x'^d{X,g{W^{Ri),W^{R2))) < min(D*(i?i), L)*(i?2)) and the 
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inequality is strict for most cases of interest. 

Now we shall study the quadratic Gaussian case to get a concrete feeling about the IPPR 
scheme. Suppose X ~ A/'(0,o-|), Yi = X + Ni, Ni ~ A/'(0,a^J, i = 1,2. Here X,Ni and 
N2 are all independent. Let Wi = Yi + Ti, W2 = Y2 + T2, where Ti ~ Af{0, a^J, i = 1,2, are 
independent of X, Ni and N2. 




R,ppr(D, 




Fig. 2. IPPR scheme 



By the IPPR scheme, for any (i?i, R2) G TZippr^Di, D2, D3), we have {Ri, R2, Di, D2, D^) G 
Q, where 

njppR{D,,D2,D3)= [j 7^'(4,4J, 

(4i'42)eS'(£'l,D2,D3) 



Tl'{ai , aj 



'Ti' "T2/ 



{R,,R2):Ri>I{Yi;Wi) = -\og 



R2>I{Y2;W2) = -\og 






cr. 



T2 
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and 



D, > E{x - E{x\w,)r =(^ + -r^—2-^ 

D3 > E(X - E{X\Wu W2)Y =(^+ 2 ^ 2 + 



(t| ^2,^ + 4^ a2^^ + 4^ 



It was computed in [15] that 

1, a 



R{D,af,) = -\og 



X 



2'-^ Dal-alal + Dal' 

where R{D, aj^-) the sum-rate distortion function of the one-encoder quadratic Gaussian CEO 

problem^. So we have 

n^rp^ ^xexp(-2i?,) + aia^^ 

Vi[Ki) = ^— 2 ) t=i,Z. 

It is easy to check that (i?i,i?2) e nippR{Dl{Ri),Dl{R2),D^) given D3 > (1/D*(i?i) + 
1/D2(-R2) — 1/4)^- Hence IPPR scheme can indeed achieve Dl{Ri) and -D|(-R2) at decoder 
1 and decoder 2 respectively in the quadratic Gaussian case. 

Let n{D^) = {(i?i,i?2) : (i?i, i?2, f^i, f^l, ^3) e Q}- It was shown in [16], [17] that 

(<72 ,<72 )GS"(D3) 



where 



^"(4,,4J = ((^1,^2) : Ri > I{YuWi\W2) = ilo; 



1, K + ^1/1 + OK + ^1^2 + 4)-^^ 



44 + ^4 + 442 



i?2>/(V'2;W^2|W^l) = -log 



1 ^ (4 + cT^i + 4)(4 + O + 4^) - ^i 

r> & 2 2 I 2 2 I 2 2 ' 



'X"T2 "T ^Afi^T2 "T "Ti"T2 



Ri + R2> I{Yu Y2; Wi, W2) = - log ^ ^ ^^ ^''^ ^ "^^ ^'' ^ 



2 ""° '^Ti'^Ti 



and 



S"(D3) = i K-O ■ D3 > E{X-E{X\W^,W2))' =(^ + ^ 



+ 



+ 4 ^^2 + 4 



^The one-encoder CEO problem is the same as the problem of lossy source coding with noisy observations 
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Let nippR{D-i) = nippR{o\,a\,D-i). For comparison, we plot TZipppiDs) and TZiD^) 
in Fig. 2. It's clear that TZippr^D^) ^ TZ^D^). A natural question is to ask whether it is still 
possible to achieve not only D3 but also nontrivial Di and D2 when the system is operated in 

n{D,)nn'jppj,{D,). 

It has been shown in [8] [15] that TZ"(a^_^,a^J is a contra-polymatroid. Its typical shape is 
plotted in Fig. 3. The two vertices Ei and E2 of TZ"{a^^,(7^^) are of special importance, where 

E, = {I{Yi;W,),I{Y2;W2\Wi)) 



1, ^x + (^k + ^T, 1, (f^i + ^l/i + 4J(o^i + ^I/2 + 4J-^: 



X 



log 2 ) r, ^'-'S ■ ^2 ^2 I ^2 ^2 I ^2 ^2 

r^ O 99 99 99 5r-v o 9 

2 a-^a^.^ + or^^or^^ + 4i'^T2 2 af^ 

Roughly speaking, the operational meaning for Ex is that encoder 1 employs the conventional 
lossy source coding and encoder 2 does the Wyner-Ziv coding; while for E2, encoder 2 employs 
the conventional lossy source coding and encoder 1 does the Wyner-Ziv coding. The Wyner-Ziv 
coding requires random binning scheme but the conventional lossy source coding does not'^. So 
when the system is operated at E^ {% = 1, 2), the decoder i can decode the data sent by encoder 
i and achieve 



1 1 



A = E(x-E(x|iyor=( — + 2^2 ) ■ (1) 



-1 



1- /(^x + (^k+(^n 



Furthermore, for vertex Ei, we have 

Ri = nvf, Wi) = I log r" ' "^^ ' "^M . (2) 

Combining (HI) and Q, we get Ri = R{Di, cr%.). That is to say, the system can achieve D*(Ri) 
at decoder i when it is operated at Ei, i = 1,2. As shown in Fig. 3, TZ{D^) is the union of 
lZ"{o'^^, crfij. The boundary of 1Z{D^) can be divided into three pieces: A, B and C . Each point 
on A corresponds to vertex Ei of TZ"(a^^, crf^J for some ((t|.^, cr^J. Each point on C corresponds 
to vertex E2 of 7l"{a^^,a^J for some (a'^^,a^J. So when the system is operated at (i?i,i?2) 
on curve A, it can achieve Dl{Ri) at decoder 1 and at the same time achieve Dl{Ri,R2) 

*For the conventional lossy source coding, the codeword is directly revealed to the decoder, which corresponds to the trivial 
binning scheme that each bin contains only one codeword. 
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at decoder 3. Curve C is similar to Curve A with the only difference that now the system 
can achieve 1^2 (-^2) at decoder 2. This observation immediately yields the following partial 
characterization of Q: 

Let A,min = E(X - E(X|F,)))2 = {l/a\ + l/a^^J-i, 2 = 1,2, and Avnin = E(X - 
nX\Y,,Y,))f = (1/4 + l/aj,^ + l/a%X'- 
For any D^ G [/^3,min, o"!], let 

( "k _ ^A ( 1 _ 1 I j_\ ~ 2 + J- - 1 > 

\^3,min £'3 / V°^ £'2,min Ds J ' max(o-|^^ .cr^^^ ) D3 Ai.min " 

a^ = <; (^ _ ^U 1 _ ^r' , ,^ , , + ^ - ^^ < and a% < < 

^ \-Di,min D:i J \Di ai^ J ' max{o-^^ ,cr^^ ) Dj. Aj.min ^^i ^^2 

oo. Otherwise, 



^T, 



( "k 

I ^3,min 


'^k\ 

Dz ) 


( 1 


1 + 1 

£'l,min Dz 


^-D2,min 


-f) 


/^ 1 


"-i) ' 


OO, 









2 + ^ - 1 > 

t4 — 3-T + 7^ — TT- — < and air > (7%^ 

max{o-^^ ,CT^^ ) -Da Ai.min ^i ^^2 

Otherwise. 



We have 

(1) For any Di e [/^i,mm, (rx{cr%^ + ^T,)/i<^x + ^A^i + ^n)l 

{(i?i,i?2) : iRi,R2,D,,al,D^) e Q} = {(i?i,i?2) : (Ri, R2) e 11(0^), R, > R{D,,al^)} 

(2) For any D2 G [/^2,mm, c^i(c^^2 + ^n)/^^^^ + ^1/2 + ^T2)]' 

{{Ri,R2) : (i?i,i?2,4,^2,/^3) G Q} = {(i?i,i?2) : iRi,R2) e n{Ds),R2 > R{D2,alj} 

(3) For any Di G [/^i,mm, f^il^^l/i + 4j/(o"i + (^n, + 4J] and D2 e [^2,mm, f^i(f7^2 + 
^r2)/K + <+^T2)]' 

{(i?i,i?2) : (Ru R2, Di, D2, Ds) eQ} = {(i?i,i?2) : R^ > i?(A,a^J,^ = 1,2} . 

Since any rate tuple (i?i, R2) on line segment i? can be viewed as the timesharing of Fi and 
F2, it implies that when the system is operated on line segment B, it can achieve D^(Ri,R2) 
at decoder 3 and at the same time achieve nontrivial Di and D2 at decoder 1 and decoder 2, 
respectively. 

The IPPR scheme can achieve Dl(Ri) and D2{R2) at decoder 1 and decoder 2 respectively, 
but can not achieve D^{Ri, R^) at decoder 3 in general The coding scheme we described above 
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Fig. 3. The boundary of TZ{D-i) 



can achieve D^{Ri, R2) at decoder 3 (at least in the case of quadratic Gaussian CEO problem) 
and at the same time achieve nontrivial Di and D2 at decoder 1 and decoder 2, but in general 
we have Di > D*(Ri), i = 1,2. We will see that these are two extremes and there exist many 
other schemes in between. 

Like the CEO problem, the multiple description problem has been studied for years and many 
multiple description coding schemes have been proposed. Here we outline the common feature 
of the existing multiple description coding schemes: encoder i {i = 1, 2), instead of sending an 
index C^' , sends a vector, say (C}"\ Q2 ); decoder i {i = 1, 2) can only decode the C}" — part; 
decoder 3 can decode both (CJ" , C| 2 ) and (C2 1 , ^^2 2 )• Clearly, this idea is also applicable in 
the distributed source coding. Moreover, we can see that the IPPR scheme corresponds to the 
case where C| 2 ^^'^ C'2 2 ^^^ constants. 

In the next section, we propose a robust distributed coding scheme by combining the random 
binning technique and the ideas from the multiple description coding. 

IV. Main Theorems 

A. An Achievable Rate-Distortion Region 

Theorem 1: {Ri, R2, Di, D2, -D3) is achievable, if there exist random variables (f/i, U2, Wi, W2) 
jointly distributed with the generic source variables (X, Fi, Y2) such that the following properties 
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are satisfied: 

(i) (f/i, W^) ^ Fi ^ (X, F2, f/2, W2) and (f/2, W^) ^ K, ^ (X, Fi, f/i, W^i). 
(ii) (i?i, i?2) e 7^(f/l, f/2, W^i, W2), wliere 

7^(f/l, f/2, W^i, W^2) = {(i?i, R2) ■■ Ri > I{Yi; f/i) + /(Fi; W^i|f/i, f/2, W2), 

R2 > /(F2; f/2) + /(F2; W^2|f/i, f/2, W^i), 
Ri + R2> I{Yu Ui) + I{Y2; f/2) + /(Fi, F2; W^i, 1^2|f/i, f/2)} • 

(iii) Tliere exist functions /^ : Wj ^ A" (i = 1, 2) and /a : Wi x Wi x W2 x W2 -> Af 
such that Erf(X,Xi) < Di,i = 1,2,3, where Xi = /i(f/i), X2 = /2(f/2) and X3 = 

MUuWi,U2,W2). 

If C denotes the set of these achievable quintuples, then time sharing yields that Qj„ = conv{C) 
is also an achievable region. 

Proof: See Appendix I. ■ 

Remark: 

1) Cardinality bound: By invoking the support lemma [41, pp.310], Ui must have |3^i| — 1 
letters to preserve the probability distribution P{yi) and 5 more to preserve /(Fi; f/i) + 
/(Fi; l^ilf/i, f/2, W2), I{Y2; W2\Ui, f/2, Wi), I{Yi; Ui) + /(Fi, F2; W^, W2\Ui, f/2), D^ and 
D3, so \V(i\ = 13^1 1 +4 suffices. Wi must have |3^i | |Wi | — 1 letters to preserve the probability 
distribution P{yi, ui) and 4 more to preserve /(Fi; Wi\Ui, f/2, 14^2), IiY2; W2\Ui, f/2, W^i), 
/(Fi,F2;W^i,lV'2|f/i,f/2) and D3. Thus it suffices to have |>Vi| = |3^i||Wi| - 1 +4 = 
|3^i|' + 4|3^i| + 3. Similarly, we have IW2I = |3^2| + 4, IW2I = |3^2p + 4|3^2| + 3. 

2) It's easy to check that TZ{Ui, f/2, Wi, W2) is a contra-polymatroid. See [42] for the defintion 
of contra-polymatroid and [8], [15], [17], [43]-[45] for its applications in information 
theory. 

3) Let WI = {Ui, Wi), 2 = 1,2. It's easy to check that 7^(f/l, f/2, W^, W^) = 7^(f/l, f/2, W^, W2) 
and X3 = /3(f/i, f/2, W\, W2) = fsiWl, W2). So there is no loss of generality to assume 
Ui-^Wi--> Yi and define f^ on Wi x W2. 

4) Let 7^'(f/l, f/2, 1^1,1^2) = {{Ri,R2) ■■ Ri > IiY^■,U^) + IiY^■,W^\U^,U2,W2),R2 > 
I{Y2;U2) + I{Y2;W2\Ui,U2)}, n"{UuU2,Wi,W2) = {(i?i,i?2) : Ri > I{Y,;Ui) + 
I(Yi- Wi\Ui, f/2), R2 > /(F2; f/2)+/(F2; W2\Ui, f/2, Wi)}. Since conv{n'{Ui, f/2, W^i, 1^2)U 
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7^"(f/l, f/2, W^i, W2)) = 7^(f/l, f/2, Wi, W2), we can let (i?i, R^) e 7^'(f/l, f/2, W^i, W^s) U 
7l"{Ui, U2,Wi,W2) in property (ii) without affecting Qj„. 

A counter example constructed by Korner and Marton [46] shows that conv{C) ^ Q in general. 
Actually even some special cases of our problem such as the multiple description problem and 
the CEO problem are the open problems of long standing. But for the following case, a stronger 
assertion can be made. 

Corollary 1: For any Di and D3, we have min{_Ri : 3R2 such that (i?i, R2, Di, -Dmax, -D3) G 
0} = min[/(Fi; Ui) + I(Yi; Wi\Y2, Ui)], where the minimization is over the set of all random 
variables (f/i, Wi) jointly distributed with the generic source variables (X, Yi, Y2) such that the 
following conditions are satisfied: 

(i) {U^,W^)^Y^^{X,Y2). 
(ii) There exist functions f : Ui ^ X, g : Ui x Wi x y2 ^ X such that Ed{X, f(Ui)) < Di, 

m{X,g{Y2,Ur,W^))<D^. 
(iii) \Ur\ = m+2,\Wi\ = {\yi\ + lf. 

Remark: For the same reason as before, there is no loss of optimality to assume Ui -^ Wi -^ Yi 
and define g on Wi x 3^2- 

Proof: Since here we are only interested in minimizing Ri under the distortion constraints 
Di and D^, there is no loss of generality to assume that R2 is large enough so that {Y2{t)}'^-^ can 
be recovered losslessly at decoder 3. In the case, our problem becomes the "noisy" Heegard- 
Berger problem. Its direct coding theorem can be easily reduced from Theorem 1 while the 
converse coding theorem can be proved along the same line as the converse in [47]. ■ 

B. Distributed Source Coding with Identical Encoders 

For many applications, it is preferable to have encoders with identical functionalities. It is 
thus interesting to study the distributed source coding system with identical encoders. In order 
to have fj^{ = y^^^ two necessary conditions are required: (1) Ri = R2, (2) 3^i = 3^2- We need 
the first condition to guarantee the range cardinalities of /]j"{ and /^ 2 ^^^ the same, and the 
second condition to guarantee these two encoding functions are defined on the same domain. 
Without loss of generality, we can assume the second condition is satisfied since we can let 
3^ = 3^1 U 3^2 and extend the probability distribution P(x, yi, 1/2) to be defined on A' x 3^ x 3^. 
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But even with these two conditions, it can not be guaranteed that the resulting encoding 
functions f^[ and /^ 2 ^^ Theorem 1 are identical. Intuitively, in order to minimize the distortion 
Ds for the fixed rate constraints, the /]j"{ and f^l generated by the random coding argument in 
the proof of Theorem 1 should be complementary to each other instead of being identical. We 
may imagine that a restriction on the identicalness of /jj"| and f^'^ may incur performance loss. 
But we will show that for many cases, no performance degradation will be caused. Firstly, we 
need a formal definition. 

Definition 3: The quadruple (R, Di, D2, D3) is called achievable with identical encoders, if 
for any e > 0, there exists an uq such that for all n > uq there exist an encoding function: 

fP : y^ -^ C(") log |C(") \<n{R + e) 

and decoders: 

IdI ■ C^"^ X C^") -^ X"^ 

such that for Xf = /^(/^"^(Fr)), ^ = 1, 2, and for ±1 = A^k/i"^!^"), /i"^(>T)), 

1 "" 

-EYd{X{t),X,{t))<D, + t « = 1,2,3. 

^ ti 

Let Q denote the set of all achievable quadruples. 

It is clear by definition that min{i? : (R, Di, D2, D3) E Q} is lower bounded by min{_R : 
{R, R, Di, D2, -D3) G Q}. The following theorem provides an upper bound on min{i? : {R, Di, D2, -D3) G 

Q}- 

Theorem 2: For any feasible^ {Di, D2, D^), 

(1) min{i?: {R, D,, D2, D^) E Q} < mm{mm{R, + R2 : (Ri, R2, D,, D2, D^) E Q},H^.,), 
where H^,, = ma.x{H{Y^), H{Y2)). 

(2) min{i? : (R, D^, D2, D^) E Q} = mm{R : (R, R, Di, D2, D^) E Q} if P(ri = y) ^ 
-P(^2 = y) for some y E y. 

Proof: (1) For any /g and /g , let /J?) = (/g, /g). So if l/gl = 2"«S l/^-^l = 2««^ 
then we have |/i;"^| < 2'<^^+^^). It's clear that if we replace both /g and /^"^ by f^^\ no 

^We say {Di, 02,03) is feasible if {{H{Yi),H(Y2), 01,02,03) G Q}. 
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additional estimation distortion will be incurred. Hence we have min{i? : {R, Di, D2, D^) E 
Q} < mm{Ri + R2 : (i?i, R2, D^, D2, D3) G Q}. 

On the other hand, for any e > 0, we can find a universal lossless source encoding function 
with rate R < iJ^ax + e that works for both {Yi{t)}'^-^ and {Y2{t)}^^. This yields min{i? : 

(2) The above proof essentially constructed a common encoder by combining two encoding 
functions. We now show that if P(Fi = y) 7^ -P(^2 = v) for some y E y, i.e., {Fi(t)}^^ and 
{^2(i)}i^i are distinguishable, we can combine two encoding functions in a more efficient way. 

For 5 > 0, let Try , be the set of 5-typical Yi-vectors with length n. Try^, is similarly defined. 
If p(Yi =y) ^ P{Y2 = y) for some y Ey, then T^^^j^ H ^[ya], = when 6 is small enough. 
Note: here 5 does not depend on n. For any e > and any R such that (i?, R, Di, D2, -D3) E Q, 
by Definition 1, there exist two encoding functions: fj^[ : y' -^ C}" and /^ 2 • 3^" ^ ^2 with 
(log IQ \)/n < R + e, i = 1,2. Here we make n arbitrarily large via concatenation. Without loss 
of generality, we assume Cj"^ = Cf'^ = C^") and (log |C(")|)/n = i? + e. Define /^"^ : 3^" -^ C^") 
such that 

Since P{Yr ^ T^^^,^ or F^" ^ T^^,],) < i^(n" ^ T^fy,],) + ^(^^2" ^ T^y,],) = <n) ^ as 
n ^ 00, if we replace both f^[ and fj^^ by /^" , the additional estimation distortion it may 
incur is at most e{n)djj^i^^, which is negligible when n is large enough. ■ 

The above proof essentially suggests a way to convert a distributed source coding system with 
different encoders to a system with identical encoders. We can conclude that for a distributed 
source coding system, we can use identical encoders^ and still achieve optimal rate-distortion 
tradeoff when the marginal distributions of the observations are different. But If P(Yi = y) = 
P{Y2 = y) for all y E y, then the restriction /^"j = /^ 2 will cause performance loss in general. 
The simplest example is to set Yi = Y2 = X. Now if we let fj^[ = y^^, then no diversity gain 
can be achieved at decoder 3. 

^possibly at the price of high complexity. 
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C. Multiple Description with Noisy Observations 

If there exist /i and /2 such that Y = fi{Yi) = /2(^2) with probability one and X -^ Y ^ 
(Yi, Y2) , our problem becomes the multiple description problem with noisy observations. In this 
case, we can directly adopt the multiple description coding scheme with only a slight change. 

Theorem 3: (1) (_Ri, R2, -Di, D2, D^) is achievable if there exist random variables {Xq, Xi, X2, -^3) 
jointly distributed with the generic source variables (X, Y) such that the following properties 
are satisfied: 

(i) X-^Y^{Xo,Xi,X2,Xs), 
(ii) R, + R2> 2I{Y;Xo) + HXi;X2\Xo) + I{Y;X,,X2,X3\Xo), R^ > I{Y-Xo,X,), 

i = l,2. 
(iii) Ed{X,Xi) <Di,i= 1,2,3. 

If C denotes the set of these achievable quintuples, then time sharing yields that conv{C') 
is also an achievable region. 
(2) Let C* denote the subset of C containing all those quintuples satisfying (i)-(iii), with the 
additional conditions that (a) Xi and X2 are independent, (b) Xq is a constant. Let 

R*{D)= min IiY;X), 

X:X^Y-*X, 
Ed(X,X)<D 

which is the rate-distortion function with noisy observations [39] [40]. Let 

QiDs) = {{R,,R2,D,,D2,D3)eQ:Ri + R2 = R*{D,)}, 
conv{C'){D^) = {{RuR2,Di,D2,D3)econv{C'):Ri + R2 = R*{D3)}. 

We have 

QiD^) = conv{C'){D,) 

Proof: Part (1) of the theorem follows from Markov lemma and Theorem 1 (specialized 
to 2-encoder case) in [29]. Part (2) of the theorem can be proved via a "continuity" argument 
similar to that of [23] by replacing Shannon's rate distortion function R^D^) with R*(D3) and 
noticing the following Markov relation: X{t) -^ Y{t) -^ iY{\Y^') -^ (/i;"l(>^i''), /£2(^2")) ^ 

Theorem[T]is associated with a distributed source coding scheme while Theorem|3]is associated 
with a centralized source coding scheme. Here "distributed" and "centralized" are in the statistical 
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sense instead of geographical sense. Even for the centralized coding scheme, we can put two 
encoders as far as possible as long as long as the inputs of these two encoders are the same. Since 
these two encoders have the same inputs, one knows exactly the operation the other will take 
and thus they can have arbitrary cooperation. In this sense, the encoders in a centralized coding 
system should be viewed as the different functionalities of a single encoder, no matter how far 
away they are separated. For a distributed coding system, since two encoders have different 
inputs, one does not know for sure about the operation the other will take. Hence, the types of 
cooperation between two encoders in a statistically distributed system are very limited. On the 
other hand, since centralized coding system is a special case of distributed coding system, one 
would expect a unified approach to both of them. But it is easy check that Theorem [H when 
particularized to the centralized case (i.e., Yi = Y2 = Y with probability one), does not coincide 
with Theorem |3l That is to say. Theorem |3] is not a "centralized" version of Theorem [B Now 
a natural question arises: Does there exist a distributed source coding scheme which subsumes 
the centralized source coding scheme in Theorem |3] as a special case ? 

Now we suggest a unified approach which incorporates these two schemes in a single frame- 
work. The main ingredient is a concept called the common part(/information) of two dependent 
random variables in the sense of Gacs and Komer [48] and Witsenhausen [49]. The following 
definition is from [50]. 

Definition 4: The common part Z of two random variables Yi and I2 is defined by finding 
the maximum integer k such that there exist functions / : 3^i -^ {1,2,--- ^k} and (7 : 3^2 ^ 
{1, 2, ■ ■ ■ , k} with P(/(ri) = z) > 0, P{g{Y2) = z) > 0, 2 = 1, 2, ■ ■ ■ , fc, such that /(Fi) = giY^) 
with probability one and then defining Z = f{Yi) (= g{Y2)). 

With this definition, it is obvious that encoder 1 and encoder 2 can agree on the value of Z 
with probability one. Therefore, they can use efficient centralized coding scheme (of Theorem 
3 type) for the common part Z and then superimpose a distributed coding scheme (of Theorem 
1 type). This observation immediately leads to the following theorem. 

Theorem 4: Let Z be the common part of Fi and Y2. (Ri, R2, Di, D2, D^) is achievable if 
there exist random variables (f/i, U2, Wi, W2, Zq, Zi, Z2, Z3) jointly distributed with the generic 
source variables {X, Yi,Y2, Z) such that the following properties are satisfied: 

(i) (x,Fi,r2)^^^ (^0,^1,^2); 

(ii) [/i ^ (Fi, Zo, Zi) ^ (X, Y2, Z2, U2) and U2 ^ (Fi, Zq, Z2) ^ (X, Fi, Zi, U^)- 
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(iii) Z3 -> (Z, Zo, Zi, Z2) ^ (X, Fi, ^2, t/i, f/2); 

(iv) W^i ^ (yi,Zo,Zi,Z2,Z3,f/l) ^ (X,y2,f/2,W^2) and 1^2 ^ (F2,^0,^l,^2,^3,f/2) ^ 

(X,Fi,f/i,iyi); 

(V) 

/?i > /(ri;Zo,Zi,f/i)+/(ri;iyi|Zo,Zi,Z2,Z3,f/i,f/2,H^2) 

i?2 > 1(^2; ^0, ^2, f/2)+/(1^2; 1^21^0, ^1,^2, ^3, f/l,f/2,l^l) 

R1 + R2 > I{Y,;Zo,Z,,U,)+I{Y2;Zo,Z,,U2)+IiZ,;Z2\Zo) 

+I{Z; Zu Z2, Zs\Zo) + I{Yu Y2; Wu W^2|^o, ^1, ^2, ^3, f/i, ^^a). 

(iv) There exist functions: fi : Vd ^ X, i = 1,2, and /a : Wi x Wi x W2 x W2 ^ A* 

such that Ed{X,Xi) < Di, i = 1,2,3, where Xi = fi{Ui), X2 = f2{U2) and X3 = 

/3(t/l,W^l,f/2,1^2). 

If C" denotes the set of these achievable quintuples, then time sharing yields that conv{C") is 
also an achievable region. 

Proof: The proof is omitted since it's a straightforward combination of Theorem [l] and 
Theorem 13 ■ 

Remark: 

1) Theorem m can be reduced to Theorem [T] by letting {Zq, Zi, Z2, Z3) = constant. If X ^ 
Z — > (Yi, Y2), then Theorem|4lcan be specialized to Theorem|3]by setting (f/i, U2, Wi, W2) = 
constant and noticing there is no loss of generality to let Zi, Z2, Z3 assume values in X. 

2) The conventional distributed source coding scheme [4] [5] does not consider the common 
part (even it does exist) of the observations and thus requires very restricted long Markov 
chain conditions on the auxiliary random variables. As we have seen in Theorem |4l the 
long Markov chain conditions are not always necessary, at least in the case when there 
exists a common part in two observations. 

3) Theorem IH essentially suggests an approach to bridging the distributed source coding 
scheme and the centralized source coding scheme. But for many cases, no common part 
exists for Yi and Y2 even when they are highly correlated. Hence it is of special interest 
to see whether there exists a general coding scheme that can transit smoothly from a 
distributed scheme to a centralized scheme when Yi and Y2 become more and more 
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correlated but no common part exists. 

V. Gaussian Case 

In this section, we apply the general results obtained in the previous section to analyze the 
Gaussian case with squared distortion measure. Although most of the results in Section IV are 
proved for the finite alphabet case with bounded distortion measure, they can be extended to the 
Gaussian case with squared distortion measure by standard techniques [7] [51]. 

Let {X{t),Yi{t) = X{t) + Ni{t),Y2{t) = X{t) + N2(t)}^^ be i.i.d. zero-mean Gaussian 
vectors such that X{t), Ni(t) and A^2(^) are independent with variances a'j^,aj^_^ and a%^ 
respectively. Without loss of generality, we only study the region {{Ri, R2, Di, D2, D^) E Q : 
-D3 < min(Di, D2), Di-iain ^ Di < aj^, i = 1,2, 3}. For convenience, we shall abuse the notation 
and denote this region by Q. 

A. An Inner Bound of the Rate Distortion Region 

We derive the inner bound of the rate distortion region for the Gaussian case by evaluating 
Thereom [1] Let Wi,Ui,W2,U2 be the auxiliary random variables jointly distributed with the 
generic source variables X,Yi,Y2 such that 

f/2 = F2 + T21 [ W2 = Y2+ T22 

Here Tn, T12, T21, T22 are zero-mean Gaussian random variables with variances a^_^_^ , a^_^^ , a^^_^ , a^^^ 
respectively, and they are independent of X, Yi,Y2. Moreover, Tn, T12 are independent of T21, T22. 
The correlation coefficient of Tn and Tj2 is Pt,, ^ = 1, 2. 
Let W* = E{Yi\Ui, Wi), i = 1,2. It is easy to verify that 

7^(f/l, U2, w,, W2) = n{u,, U2, w*, w;), 

E{X - E{X\Wi, W2, Ui, U2)f = E(X - E(X|Vr*, W^, Ui, f/2))^ = E(X - E{X\W^, W;)f. 

So there is no loss of generality to assume Ui ^ Wi ^ Yi, i = 1,2, i.e., we can assume 
Til = Ti2 + ATi, 1 = 1,2, where ATi ~ U{(), 4^^ - a^J and ATi ~ Ar(0, 4^^ - 4^ J are 
mutually independent, and they are independent of X, Yi,Y2, T12 and T22. 
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Now by evaluating Theorem [H we get the following achievable rate distortion region: 



Qin = conv 



y(4n^4i2'42i>422) 



where 



/7/ 2 2 2 2 ^ 

l- y^Tii 5 '^Ti2 ' ^T21 ' '^T22 / 
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B. An Outer Bound of the Rate Distortion Region 

Let e{t) = X{t) - S{t), t = 1, 2, • ■ ■ , where 

6(t) is Gaussian with mean and variance -Da^min, and is independent of Yi{t) and V2(t). Let 
dx = (^\- ^3,min and di = Di - L's^min, ^ = 1,2, 3. Define 



where 



Qout= [J Cout{ru,ri2,r2i,r22) 

(rii ,ri2 ,r2i ,r22)eSoiit 



1 exp(2rji) 



Co«t(rii, ri2, r2i, r22) = < (i?i, i?2, ^i, ^2, ^3) : tt < o ' ^« ^ ^^i + ^^2, ^ = 1,2, 



/^,: - fT 



2 ' 



^ ^ 1 - exp(-2ri2) , 1 - exp(-2r22) 

7^ — o ~r 



\{Di, D2, D^,r 21, r 22) = < 



D3 crl a%^ al^ 

ru + r2i > - log -y + A(i:)i, D2, ^'a, ^21, r22) \ , 



0, C < di + d2 - dx 

llog^^ C> ("- + --- 

2 ^'^& did2' ^ - \^dl ^ d2 dj, 



2 



Mog ; , y X Q — otherwise, 

2 ^ (rfx-C)2-[V(rfx-dl)(rfx-rf2)- V((il-C)(d2-C)]2 



^ _ n n / exp(-2r2i) exp(-2r22) 

(, — i^s-L^s^min I 2 ' 



^JVl ^iV2 



and 



^out = \ {rii,ri2, ^21,^22) G M^ : ^- exp(2rji) < ^- H 2 ~^ i = 1,2 

Theorem 5: Q C Qo«t- 

Proof: The proof is left to Appendix IL ■ 

It's easy to check that {(Ri, R2, D^) : (Ri, R2,ax,a'j^, D^) E Qout] is the rate distortion 
region of the quadratic Gaussian CEO problem [17] and Qout converges to the rate distortion 
region of the Gaussian multiple description problem [20] as (T^^jCt^^ ~^ 0- Moreover, Q^ and 
Qout coincide in some subregions as shown in the following corollary. 

Corollary 2: For any (i?i, i?2, ^1, ^2, £'3) G Q, if Ri = R{Di,ajjJ, i = 1,2, then I/D3 < 
1/Di + I/D2 - II o\. 
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Proof: From the outer bound, we have Ri > rn + rj2, Di > a\ exp(— 2rji), and 1/cf\ + 
(1 - exp(-2ri2))/o-^^ > exp(2rji)/o-|, i = 1, 2. Thus if Ri = R{Di, a%J, i = 1,2, then 

r.i = ^hg-, 

ri2 = - log 



2 '"° Dial - a\al^ + D^a^^^ 



Therefore, we have 



1 ^ _}_, 1 - exp(-2ri2) 1 - exp(-2r22) 



_ 1 1 1 



It's easy to check that IPPR scheme can achieve all the (i?i, R2, Di, D2, -D3) satisfying Ri = 
R{Di, cfI), i = 1,2, and I/-D3 < I/-D1 + I/-D2 — V^x- Hence for the quadratic Gaussian case, 
it's impossible to have D^^ smaller than that achieved by the IPPR scheme if decoder 1 and 
decoder 2 are rate-distortion optimal. 

C. A Symmetric Case 

.2 _ ^2 _ „2 



Let a^i = at = aj^. It was computed in [15] that 



2 



R = Uo-^ ^^ ^ 2D|(i?,i?V^ 



4 " \ /}*(/?, R) \2D*iR, R)al - a^aj, + D*iR, R)aj, 
or equivalently. 



2ala% + a\a% + 2a\ exp(-4i?) + 2 a\ exp(-2i?) y/g^ exp(-4i?) + 2a\al +^ 

2 I ^2 \2 



'' ' ' (2^1 + ^^ 

Define r'i2(i?) = {(£>!, ^^2) : {R, R, Di, D2, D*{R, R)) G Q}. We are not able to give a 
complete characterization of Vi2{R). Instead, we shall establish an inner bound and an outer 
bound. Let 

VT2{R) = {{Di,D2) : {R,R,D^,D2,Dl{R,R)) G Q.„} 

and 

V^^R) = {{D,,D2) : {R,R,D,,D2,D;{R,R)) G Qout}- 
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Since Q^^ ^ Q C Q^^^, it follows that V'{;^{R) C Vi2{R) C V^l^^R). Now we proceed to 
compute the explicit expressions of 'D^^(R) and 'D°2^(R). 

It can be seen that {R, R, Di, D2, D^iR, R)) G Qout if and only if there exist (rn, ri2, r2i, r22) G 
M^ such that the following set of inequalities are satisfied: 

R > ra + ri2, (3) 

D, > 4exp(-2r,i), (4) 

exp(2r.) < i + ^i^^Wz^, , = 1,2, (5) 

rii + r2i > - log ^, !^^ ^. + A(Di, D2, D3, r2i, r22), (6) 

1 1 l-2exp(-2ri2) l-2exp(-r22) 

— 9~r 9 ~r 9 • V'/ 



By [45, Lemma 3.5], if {ru,ri2,r2i,r22) G M^ satisfy this set of inequalities, then we must 
have 

and 

_ 1 / 2D*{R,R)al \ 

ri2 - ^22 - 2 og [2D*{R, R)al - aj,a% + D*,{R, R)a% ) ' ^^^ 

From equation Q, it is easy to get that 

C\d,=dur,r) = d;{r,r)-(\ + ^) . (10) 

Equations ® and © implie that 

A(Di,D2,D3,r2i,r22) = 0, (11) 

which further implies that CI 1)3=01 (ij.i?) < rfi + ^2 — (ix, i.e., 

Di + D2> C\ds=dur,R) +^x+(\ + ^) = D;{R, R) + a\. (12) 

By @ and ©, we have 

r„ < ■nm(ilog(l + "^""y-^'--^' ).fl^,,. 



1,„ ffl: + CJ(fi,iJ) 1,_ ffi 



min - loe ; — , - loe — , 

\2 ^ 2D*{R,R) '4 ^D*{R,R) 
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which, together with (JH), implies 



1, ^i 

ru = rai = - log ■ 



4 ^D*,{R,Ry 
Thus by ©, we obtain 



A > 4exp(-2r,i) = axVD*siR,R), ^ = 1,2. (13) 

Combining dHl) and dOl) yields 

I)°f (i?) = 1^1,^2) ■■D, + D2> D;{R,R) + al,D, > ax^ Dl{R,R),i = 1,2} . (14) 

The main technical difficulty of computing V^^^R) lies in the convexification operation. 
Fortunately, the following lemma significantly reduces the computational complexity. 

Lemma 1: For any Ai, As G (0, 1) with A1 + A2 = 1, and (i?;, i?^), (/?'/, R'^) with \i{R[, i?^) + 

A2(i?i, i?2) = {R^ R)^ we have XiD;{R[, R!^)+X2Dl{R!{, R!^) > D;{R, R) i{R[+R'^ ^ R'l+R^ 

Proof: See Appendix III. ■ 

This lemma implies that it is impossible to achieve D^(R, R) by timesharing two distributed 
source coding schemes, one with the sum-rate higher than 2R and the other with the sum-rate 
lower than 2R. Therefore, we have 

Pi^(i?) = |(A,/^2) : {R,R,DuD2) e conv (|J (^1 U ^2)) } , 



where 



A = <! (i?i,i?2, A, A) : ^ < ::^ + 2 !^ 2 >^ = 1'2, 



Di crj. ai + a, 



il 



-^1 = ^°S 2 r 2 2 IT' -Rl + -R2 = 2i? 



1 1 



A = (i?i,i?2,/^i,/^2) : 77 < — + 2 ^ 2 >^ = 1'2, 

-Ri = o ^°S 2 / 2 — 2 TT' -Ri + -R2 = 2i? 

and [J is taken over all {<Jt^^ , a^_^^ , a^^_^ , crf^^i ) ^^^^ that a^^^ > crf.^^ ' '^T2i — ^^21 ^^'^ 

2R = ^log t1-^"^-"t-"1l (15) 



2 <2<2(«-<^ 

1 1 1 

ri2 '^A^ "^ "^Taa 



D3*(i?,i?) 4 ^^^ + 4.2^^^ + ^! ■ ^^^^ 
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Let 



^* = 7^ log 



1 ADl{R,R)a 



[al + Dl{R, R)] [2D*,{R, R)al - alcr% + D*,{R, R)a 



IV 



^{x) 



[2D*{R, R)a\ - a\al + D|(i?, R)al] exp [2{2R - x)] - 2D|(i?, R)a\ 



a\-Dl{R,R) 
Note dni) and (El) imply that 

2a\Dl{R, R) - alal + alDl{R, R) 



CFrr,„ = an 



and cr^^^cr^^^ = oo (i.e., a^_^^ = oo or crf.^^ = oo). Therefore, 

P™(i?) = |(Di, D2) : {R, Di, D2) e conv (Ii U I2) } , 

where 

A = UR,Di,D2):R*<R<2R-R*,Di>ip{R),D2 = al\, 

Ai = UR,Di,D2):R*<R<2R-R*,Di = al,D2>ipi2R-R)\. 

LotdVr2{R) = {{Di,D2) G Vr^{R) : {D[ < Di,D^ < D2) => {D[ = Di,D^ = D2),V(D;,D^) G 
r''i^(i?)}. It is clear that Vl'^^iR) is completely characterized by dV%{R). Note that A U ^2 
is a subset of a 3-dimensional linear space. Thus by Caratheodory's fundamental theorem [52], 
for any {Di, D2) E d'D'^(R), (R, Di, D2) can be expressed as a convex combination of at 
most 4 points in Ai U A2- Actually this can be further simplified. Since ip{x) is a convex 
function, it implies that for any (Di, D2) E d'D'^{R), {R, Di, D2) can be expressed as a convex 
combination of a point (i?', D[, D'2) G Ai with D[ = ip{R') and a point {R", D'{, D2) G A2 with 
D'l = Lp{2R — R"). Now the problem is readily solved by Lagrangian optimization. Through 
tedious but straightforward calculation, dV^'^iR) is the curve Di = i'{D2) given by the following 
parametric form: 

D^ = ,/~T aj. + ,/~f ip(2R - R*) 

2R^R'~^. X 2R~R'-f.^y J for i?* < /i < R, 

D2 = d^vi^R - ^) + 2«^^i 

^-^ X R*-^,Y\^-) for R<n<2R-R*. 



D2 = §^^^{2R - R*) + f5f c^i 



Hence we have 



VT2iR) = < (^1, D2) : D, > ^(D2), A > ^^^^^^^1^ , ^ = 1, 2 
As we can see in Fig. 4, T>^2\R) is strictly bigger than Vl'^lR). 
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D?^'(R) 



Fig. 4. Compai-ison of Vi':^{R) and V1f{R) 



D. An Extreme Case 

Theorem 6: Let Q^ = {(i?i, £>!, £'2, £'3) : (^1, 00, L>i, L>2, ^3) e Q}. We have (i?i, L)i, D2, D^) e 
Qe if and only if D2 > -D2,min and 



Ri > 



^log- 



4 4 



■<2 +Dl) (D3^1:^^^ +£'3^1:^ir2 +03<Tir^ <Tir2 -'^l:<Tir^ ^^2 ) ■ 



^(A,<; 



D.< 



D.> 






Proof: Since R2 = c)o, we can assume that {Y2{t)}'^^ is directly present at decoder 
2 and decoder 3. Hence any D2 > -D2,min is achievable. Now only (Ri, Di, D^) remain to be 
characterized. The achievability part follows directly by evaluating Qj„ with a^^^ = a^^^ = 0. For 
the converse, it is clear that Ri > R{Di, cr|^J, which resolves the case D^ > Diajf,J{Di + cr^J. 
For the case D3 < Dia%^/(Di + crl^J, the details are left to Appendix IV. ■ 

Remark: The converse can not be reduced from Qout, which shows that our outer bound in not 
tight. 

Theorem |6l implies that 

and min{Di : (_Ri, 00, Di, D2, Dl{Ri, 00)) E Q} = aj^ for D2 > -D2,mm- That is to say, for this 
extreme case, if decoder 3 achieves the minimum D3 for a given Ri, then it is impossible for 
decoder 1 to make a nontrivial estimation of {X{t)}^^. 
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E. Noisy Multiple Description for the Gaussian Case 

Now consider the case when both encoder 1 and encoder 2 can observe {Fi(t)}^^ and 
{^2(^)}i^i simultaneously. Clearly, the rate distortion region of this problem (which we denote 
by Q!) is an outer bound of Q. 

If we assume encoder 1 and encoder 2 can only observe {E(X(t)|Yi(t), Y2{t))}'^^, and let Q" 
be the rate distortion region for this case, then clearly we have Q" C Q' since {E(X(t) |Fi(t), V2(t))} 
can be computed from {Yi(t)}^i and {Fal^)}^!- 

Theorem 7: Q! = Q" = T, where 

r = I (i?i, i?2, Di, D2, Ds) : i?i + i?2 > ^ log ^ + 7(rfi, d2, 4), R^>l log ^,2 = 1,2 

and 

0, ds < di + d2 - dx 

^{d„d2,d,)=l ilogff, ^3>(i + i-^ 

i log , ^ ^~ ^' — , , Otherwise. 

Proo/.- As defined before, let S{t) = E{X{t)\Yi{t),Y2{t)) and ^(t) = X{t) - S{t), t = 
1, 2, ■ ■ ■ . We have E9^{t) = Dg^^im and ES^{t) = a\ - D^^^^,, = dx- 

Now we view {S'(t)}^^ as the source, and let Qs be the multiple description rate-distortion 
region for {S'(t)}^^. It was proved by Ozarow [20] that (i?i, R2, Di, D2, -D3) G Qs if and only 
if 

R3 > l-\og^ + ^s{Di,D2,D,), 

where 

'0, Ds<D, + D2- dx 



00 



75pl,/^2,/^3 



1 log; ^2dZ3 £)„ > 



2 ^"^6 D1D2' -^3 ^ \^£)j -r £)2 dx 

i log , ( X- 3) — Otherwise. 

2 ^ {dx-D3y-[y/{dx-Di)idx-D2)^y/{Di-Ds)(D2~Ds)]'' 



Since {^(t)}"=i is independent of {Yiit), F2(t)}r=i and thus is independent of {Xi(t), X2(t), X3(t)}JLi, 
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we have 



^ n 1 " 

t = l 

1 " 

^3,min + - V E(5(t) - X,(t))2, Z = 1, 2, 3. 
n -i— ^ 



n -^ — ' n 

n 



n 
t=i 

Hence (i?i, i?2, D,, D2, D,) e Q'(or Q") if and only if (i?i, i?2, /^i - /^3,min, D^ - Ds.min, ^^3 - 

^3,min) G Q5 (i.e., (i?i, i?2, c?i, c?2, 4) e Qs). The proof is complete. ■ 

Remark: Theorem |71 is still true when Ni{t) and A^2(^) are correlated (with correlation 

coefficient pn), now 

Note: Dg.min = (l/fT| + l/ffir)"^ if a%^ = a%^ = a% and pn = 1. 



VL Conclusion 

We proposed a robust distributed source coding scheme which flexibly trades off between 
system robustness and compression efficiency. The achievable rate distortion region of this 
scheme was analyzed in detail for the Gaussian case. But a complete characterization of the rate 
distortion region Q, even for the Gaussian case, remains open. We believe that the following 
problem deserves special attention. For the Gaussian case, Q\^2 ^^2 ^q is the rate distortion 
region of the multiple description problem, which has been completely characterized in [20]. 
The question is whether Q converges to Q\^2 ^^2 ^q as cr^^ and a"^^ go to zero. A solution to this 
problem will have many interesting implications and can significantly deepen our understanding 
of the multiple description problem and the distributed source coding problem. 



Appendix I 
Proof of Theorem 1 

The proof of Theorem 1 employs techniques which have already been established in the 
literature, especially in [43] [47] [53] [54]. Hence we only give a sketch here. 
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For each f/i, f/2, Wi and W2 satisfying Property (i) and (iii), we prove the admissibility of the 
rate tuple (Ri,R2), where 

R^ = I{Y^■,U,) + I{Y^■W,\U^,U2,W2), 
R2 = I{Y2;U2) + I{Y2;W2\Ui,U2). 
Then by symmetry, the rate tuple {R[, R2) with 

R[ = I{Y^■U^) + I{Y^■W^\U^,U2), 
R'2 = I{Y2■,U2) + IiY2■,W2\U^,U2,W,) 

is also admissible. It's easy to check that 

/(Fi; W,\U,, f/2, W2) + /(F2; W2\U,, f/2) 
= I{Y,■W^\U,,U2) + I{Y2■,W2\UuU2,W^) 
= I{Y,,Y2;W,,W2\Ui,U2), 

now Theorem 1 follows by timesharing (i?i,i?2) and {R'^.R^). 

It was established in [47] that for any positive e and sufficiently large n with 

|Cj")| < exp(n(/(Fi; f/i) + I{Y,- W^p^, f/2, W2) + e)), 

decoder 1 and decoder 3 can recover m" and construct £" with xi{t) = fi{ui(t)),t = 1, 2, ■ ■ ■ , n, 

such that 

-EYd{X{t),Xi{t)) <Di + e, 
n ^ — ^ 

and provided ^3 ^iid w'2 are available to decoder 3, it can further recover to" and use w", Wg ^ ^"^ ^2 
to construct £3 with £3(t) = /a {wi{t),W2{t),Ui{t),U2{t)) , t = 1, 2, ■ ■ ■ , n, such that the average 
distortion is less than or equal to D3 + e. 
Again by [47], with 

\ci"^\ < exp{n{I{Y2; W2) + 1(^2; f/2|W^i, W2) + e)), 

decoder 2 and decoder 3 can recover U2 and construct £2 with X2{t) = f2{u2{t)),t = 1, 2, ■ ■ ■ , n, 

such that 

1 " 

-Eyrf(X(t),X2(t)) <D2 + e, 

n ^ — ^ 
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and provided n" are available to decoder 3, it can further recover Wg . 

In summary, decoder i recovers m" {i = 1, 2), and decoder 3 recovers u", Ug ? ^"; ^2 with the 
decoding order {u^,U2) -^ W2 ^ w^. 

Thus we have established the admissibility of the rate tuple (i?i, R2) and completed the proof. 

Appendix II 
Derivation of the Outer bound 

Let rn = /(X";/^"J(y;"))/n,ri2 = /(F^"; /|^J(Fi")|X")/n, i = 1,2. The following lemmas 
were proved in [17] with the method developed by Oohama [14]. 
Lemma 2: 

1 fo \ ^ 1 1 - exp (2ri2) . 
^-exp(2rii) < ^-H ^ ^ = 1,2, 

-lexpg/(x";/gm/S(1^2"))) < ^.l^-P(z2n^.l^-PMr,^ 
Now we are ready to derive the outer bound. 
Proof: By data processing inequality, we have 



^1 ^L ^1^2 



I (X"; /g(Fr)) > ^ (^"; ^r) ^ i log J' ^ = 1' 2 (17) 

/ (x- /i:i(n'^)/g(>^2")) > / (^"; ^3") > I log ^. (18) 

It follows from dnl), dUl) and Lemma El that 

J_ ^ exp(2ra) . _ . ^ 

1 ^ 1 1 - exp(-2ri2) 1 - exp(-2ri2) 
^3 " (y\ a\ a\ 



Since X" ^ F^" ^ ftl^D^ z = 1, 2, we have 



n 

rii+ri2 i = l,2. 



X" 
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Now we proceed to derive a lower bound on rn + r2i, 
n{rii +r2i) 

= / (x- /g(Fr)) + 1 (^"; /S(>T)) 

/ (x- /g(F«), /^(y^")) + / (/S(n"); /g(>^2")) - / (/S(n"); /S(>^2") | ^") 
/ (x- /g(n"), /|:J(F2")) + / (/g(n"); /S(>^2")) (19) 

where (a) follows from the identity 

I {A- BC) =I{A-B) + I {A- C) + I (B; C\A) - I {B; C) . (20) 

and (b) is because Z^; "{(y") -^ X" -^ fEiO^'T)- Now applying data processing inequality, we 
have 



(a) 
(b) 



n{ru+r,,) > / (x"; X") + / (/glF,"); /g(^2") 

> ^ log ^ + / (/g(n"); ftiiy2)) ■ (21) 

To lower-bound / (/e 1(^1^); /jj 2(^2") )' ^^ introduce an auxiliary random vector Z"^ such 
that Z{t) = S{t) + M(t),t = 1, 2, ■ ■ ■ , n, where the M{t)'s are i.i.d zero-mean Gaussian random 
variables with variance a\j (which will be optimized later). We assume that M" is independent 
of (X", F", Fg")- Since 6'" is indepedent of F", Y^ and thus independent of X", X^, we have 

A > -^E(x(t)-X,(t)^ 



n 



I.e., 



Since 



1 " 

1 " 2 

= -5^E(5(t)-X,(t)) +Z^3,min, 
t=l 

ij^E (^(t) -X,(t))' < A - A3,min = c/„ ^ = 1,2. 

i5^E(z(t)-X.(t))' = l5^E(5(t)-X.(t))Vl5^EM^(t) 

< di + alj, ^ = 1,2, 
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by rate distortion theory, 



^l-'-")4-(tTf 



1,2. 



Now applying the identity (|20|) to / (/^"l(>"i"); /S(^2"))' we get 



z")-/(z-/g(n"),/g(F2^ 



> — log 

- 2 ^ 



dx + o-|^ 



dx + (^li 



di + (^M J \d2 + a-j^ 



,{n) 



-/(z";/g(y,"),/g(y,'^)). 



(22) 



We upper-bound / Z"; /rK^i"), /^ 2(^2") as follows: 



in). 



/(^";/S(n'^),/S(>^2^ 

^ log [2vre (rfx + ai,)] - h [s^ + M«|/g(n"), /g(r,^ 



< - log [27re (dx + a^^)] - 77 log ^ exp 



2 



-M51/g(n"),/g(>T)) 



+ 2nea 



M ( 5 



,(23) 



where (c) follows from the conditional version of entropy power inequality [55]. Since 



h 


is- 


h 


V 


h 


(s- 


h 


'5" 
V 



,(n) 



in), 



tEwnj^Em 



[■{")/ 



<•(«) 



x\ mrn^mY,-) + / U- X 



cin) 



x\fZ{yi).fEm 



in). 



+ / (X"; 5") - / (X"; /g(n"), /S(>^2")) , 
/S(n"),/S(>^2")),wehave 



where the last equality follows from X" — > S*" 

exp0/.(5"|/g(n"),/S(>^2"))) 
= exp (/. (5" |X", /g(n"), /g(F2") )) exp (/ (X"; S^) exp (-/ (x"; /g(n"), /S(>^2"))) 
"'' exp {h{S-\X\ /g(n"), /^(y^"))) exp (-/(X"; /g(n"), /^(F^"))) • (24) 



D 



S.niin 
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Now we shall derive a lower bound on exp ( f ^ ( 5*" X", /^ 1(^1"), /I; 2(^2") ) ) • Since condi- 
tioned on (X", fEiO^i')^ /I; 2(^2") )' ^1" ^^'^ ^2 ^^^ independent, by the conditional version of 
entropy power inequality [55], we have 



exp I —h I 5"^ 
\ n 



X-jZ^Yl)^fE2iY, 



(«), 



> 



— h ( 3,min yn 



x\fZiYnJB('^2 



^exp( 

DUn E ^ ^^P (-h (^^" ^"' /S(^")' /S(>^2") 

^Lin E 4 exp (^M>^r 1^") - §/ (Yr, fSoT 



^n 



2TieD^ 



3,min 






exp (-2r2j 



a 



Ni 



Thus by (EH) and ^, 

'2 
. n 



exp(^/.(s"|/g(rn,/g(>^2" 

2 



^ 9^,^2 n f V- exp (-2r2i^ 



, i=l 



n 



(«), 



f("), 



exp --/ X";/^"i(F,"),/g(F," 



Combining (ES) and (1261) yields that 



- ^ log ( 4^3,.in ( E ^^^^^^ J exp (-^/ (X- /g(yr), /g(>^2" 



Substitute (Ell) into (|22ll and then apply ( IT9b . 

/(/g(n");/g(>^2") 



n 



M 



(c/l +Crf^)(rf2 + 0r 



M, 



+ - log I (Tx^3,min ( 2^ 2 



exp(--/(X-/S(n"),/S(^": 



> — loer 
- 2 ^ 



ra 



(ix + cr|,^ 



2 



+ 77 log 0-j^^3, 



E 



exp (— 2r 



2«; 



cr 



A^, 



pW 



exp ( -/ (/g(rr); /S(>^2")) j exp (-2(rn + r,,)) + a 



(25) 



(26) 



+ a 



M 



+ ^M 



^1/ 
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which can be rewritten as 

exp0/(/g(rn;/S(>^2" 

> (dx + <yli) <Tm ^27) 

{d, + al) {d2 + al) - a|D3,min {dx + o\) exp (-2 (m + r^i)) f E 22±^' 



Combining (EB and (^Tl) yields that 

exp(2(rii +r2i)) 



> _ ^i^M(^i + 4/) 



D3 ( (rfi + af,) (rf2 + crl) - a\D,^^^, {dx + al) exp (-2 (m + r^i)) ( E ^^^^^^^^ 



i=l 



^X 



which can be further written as 



where 



ru + rai > - log -^ + r7(a|,^, rfi, c?2, C), 



C = i^3,..I^3|E"'^"''^^ 



,.=1 <. 



r]{aM,di,d2X) = t^^^S 



2 ^ {di + alj) {d2 + a 



Ml 



Calculus shows that 

supr7((jf^,di,d2,C) 



4. 



77(00, t/i, (^2, C) = 0, C<di + d2-dx 

r,{0,d„d2,O = llog^^, C>(i + i-i)"' 



where 

.2 _ '^ic^2 - dxC + \/{dx - di){dx - 1^2) (c?i - C)('^2 - C) 
''^~ dx + C-di-d2 
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Appendix III 
Proof of Lemma 1 

Define the functions yi = 0i(a;i) and 1/2 = 02(2^2) via tlie following parametric forms: 



1 /I l-exp(-2ai)\ 1, 2 



and 



1, /I 2-2exp(-2ai)\ 1, /I l-exp(-2ai)\ 



1 f 1 2-2exp(-2a^\ 1 /"l , 1 - exp(-2a;) , , ^ 



1, /I l-exp(-2a2)\ 1, 2 

1/2 = -log --^ + 2 +:5logf^X + "2, 

2 V^x ^jv / 2 



where ai,a2 > 0. Define Q = {(i?i,i?2) G 7^+ : 0i(-Ri) < R2 < 02(-Ri)}- It is easy to check 
that {R, R) is an interior point of ^2 for any R > 0. 

Let r denote the line segment from {R[, R'^) to (/?'/, R'^). It is clear that Dl{Ri, R2) must be 
a convex function of (i?i,i?2)- Hence we have \iDl{R[,R'^) + \2D*{R'I,R'^) > D*{R,R). If 
the equality is achieved, then it implies that D^(Ri, R2) is linear on V. 

It was computed in [15] [45] that for any (i?i, R2) eVL, 



2 



R, + R,^U,ri "^i / 2B5W.fl3K 



or equivalently, 

D*(i?l,i?2) 



2 ^ I Dl{Rr, R2) \2Dl{R^, R^)a\ - a\al + D^(i?i, i?2)a^ 



2a^^al + aK + 2ai exp(-2(i?i + R2)) 




(2^i + ^^)^ 



2a^exp(-(i?i + i?2))A/^iexp(-2(i?i + R2)) + 2aia|, + < 

(2cx| + O^ 



which is strictly convex with respect to Ri + R2. Hence if R[ + R'2 ^ R" + R", then Dl{Ri, R2) 
is strictly convex on f] n T and we must have \iDl{R[, R'^) + \2Dl{R'{, R'^) > D^{R, R). Note: 
Since (i?, R) is an interior point of ^2, 1] fl F is not empty. 
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Appendix IV 
Extreme Case 



X' 



nR, > H (/g(n")) = / (n"; /gin")) = / (x^ F^"; /g(y;" 

= / (x", F,"; /g(n"), F2") - / (^", n"; >^2" |/S(n": 

= j(x-/g(Fn,^2") +^(n^/S(n")|^") +^(n";>^2"|^">/S(n" 

-/ (X", n"; y^") + / (/g(rr); ^2") • (28) 



Now we bound each term separately. By data processing inequality, we have 



[■(")/ 



n 



a 



X 



I { X- /^^Kn"), Y-) > I (^X- X^) > - lo„ ^^ 



Applying Lemma |21 with f^iO^'T) = ^2"' we get 



1 1 l-exp(-|/(l^-/a(n" 

— I 1 ^^ ^^ '■ 

2 ' 2 ' 2 
'i o-ir air 



X' 



^1 <2 



'Ni ^X 

Combining (l29l) and (l30l) and after simple calculation, we obtain 



>-lexp(^/(x-/g(F«),r,' 

(J Y \ li \ 



/(y-/g(r- 



n 



X" > - lo 



^Wn.Ds 



2 "° {D^a\ol^ + D^a\al,^ + /^s^^.^^^ " '^Wm'^l^] 



Since y^'^ -> X" -> 1^. it follows that 

/(F-F2"|^",/S(n"))=0 



and 



n 



^\ + ^1/2 



/ (X", Fi"; r^") = / (X"; 1^2") = 7T log 2 

z o 



N2 



For the term / ( f^\] Y^ ) , since 

-J2E{Y,{t)-Mt)r = -$^i?(X(t)-Xi(t)f + -^EX,2(t) 



t=i 



t=i 



t=i 



< Di + a%^, 



(29) 



(30) 



(31) 



(32) 



(33) 



DRAFT 



CHEN ET AL. :ROBUST DISTRIBUTED SOURCE CODING 37 

by data processing inequality and then rate distortion theory, we have 

/ (/S(n"); y^) > I (A"; Y,-) > I log ^±^. (34) 

Now substituting (I29l)-(l34l) back to (EH), we get 

1 4 4 

1 ^X'^N2 

The main technical difference between the derivation here and the one we used to prove 
the outer bound in Appendix II is the way to lower bound / [fEiO^i)] je 20^2) )• Since for 
the extreme case it reduces to the problem of lower bounding / (/I; 1(^1"); ^2")' we adopt a 
straightforward approach as shown above, rather than the method of Ozarow [20]. 
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